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ABSTRACT 



This introduction to Phase II, of a four-phase study, 
provides information on the background of the California Educational 
Technology Assessment Project (CETAP) , the legislative history and rationale 
for the study, and the general evaluation plan. Phase II presents an in-depth 
formative and summative assessment of six major programs funded by AB 
1470- -the Farr-Morgan-Quackenbush Educational Technology Act of 1989. The six 
programs studied in this phase are: (1) the first year of schools receiving 

School -Based Educational Technology Grants; (2) Model Technology Schools 
(Level I); (3) Level II Model Technology School Projects; (4) Instructional 

Television Regional Agencies; (5) Software Development Projects; and (6) the 
California Technology Project and its regional consortia. Six priorities for 
the evaluation of the educational technology programs included: site-based 
technology use planning; level of implementation; curriculum support and 
alignment; staff development; learning resources management; and evaluation 
and accountability. It is noted that the data gathered for the study were 
concerned with the cost-benefits of each program; the extent to which program 
objectives and activities were implemented; the extent of integration and 
support of the curriculum frameworks; the influence of the programs on school 
planning for technology use; the coordination of projects with existing 
programs and resources; and the impact of projects on students, staff, and 
instructional programs. (Contains 78 references.) (AEF) 
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Report Overview: The first section of this report provides: 1) the background of the California 
Educational Assessment Technology Project (CETAP), 2) the legislative history and rationale for the 
study, and 3) the general evaluation plan for the study. The following six volumes each address one of 
the six programs and projects studied in Phase II of the study. Each section provides: 1 ) a detailed 
description of the program evaluated, 2) the evaluation procedures specific to that program, 3) the 
findings by project component, 4) summary and recommendations for the program and, 5) the 
instrumentation used to gather the data. 



I. Background 

The in-depth evaluation of the state’s educational technology programs was conducted by the Far West 
Laboratory for Educational Research and Development (FWL) and the American Institutes for Research 
(AIR). FWL is the prime contractor with the California Department of Education (CDE), and AIR was a 
subcontractor to FWL for this project. Specifically, FWL is responsible primarily for the following: 1) 
evaluation of fourteen selected AB 803 programs and projects; 2) evaluation of five AB 1470 programs, 
including the School-Based Educational Technology Grant program, the Level II Model Technology 
School Projects, Instructional Television (ITV), the California Technology Project (CTP), and Software 
Development; and 3) production of evaluation templates and guidelines for 1 and 2 above. AIR is 
responsible primarily for the following: 1) evaluation of the six Level I Model Technology Schools 
(MTS) Projects, 2) a cost-benefit analysis of the six programs included in Phase II of the study, 3) an 
“evaluability” analysis of all projects and programs, and 4) collaboration with the FWL staff in analysis 
and reporting of the project findings. 

The official starting date for CETAP established by the CDE was October 1, 1990. Periodic updates and 
interim findings related to this study have been provided to the CDE and the Educational Technology 
Committee. The Refinement of the CET AP Evaluation and the First Cumulative Progress Report was 
submitted and approved by the Office of Educational Technology and was also distributed to the 
Educational Technology Committee. 

This report is the second phase of a five phase study. Phase I is a descriptive study of educational 
technology projects and programs funded by AB 803 from 1984 to 1989. Phase II of CETAP provides a 
review of the in-depth formative and summative assessment of six major programs currently funded by 
AB 1470 - the Farr-Morgan-Quackenbush Educational Technology Act of 1989. These six programs 
are: 1) the first year of schools receiving School-Based Educational Technology Grants; 2) Model 
Technology Schools (Level I); 3) Level II Model Technology School Projects; 4) Instructional 
Television Regional Agencies; 5) Software Development Projects; and 6) the California Technology 
Project and its regional consortia. 

To conduct the evaluation a broad base of data was gathered to answer a wide range of questions about: 
1) cost-benefits of each program, 2) extent to which program objectives and activities were 
implemented, 3) extent of integration and support of the curriculum frameworks, 4) influence of the 
programs on school planning for technology use, 5) coordination of projects with existing programs and 
resources, and 6) impact of projects on students, staff, and instructional programs. The Phase II report 
provides a description of each program, the evaluation and data collection methodology, 
review of findings, and brief conclusions and recommendations for each of the six programs. An in- 
depth analysis of Phase II information with answers to the evaluation questions will comprise the Phase 
III report of the CETAP. 



II. Legislative Overview and Mandate for the Evaluation 



Legislative Intent of AB 1470. In approving AB 1470 the Legislature declared educational technology 
to be a valuable tool for improving and expanding instruction to meet the evolving needs of all 
California pupils, offering benefits to students in every grade, at all levels of ability and in all geographic 
regions. The legislation provided several guidelines for schools: 

• Representatives from all educational levels and from the business community should 
be involved in planning and establishing educational technology policy. 

• Instructional technology uses should be developed, implemented and evaluated at the 
school level with district, regional, state and private sector support 

• Resources should be identified and utilized which will improve staff development 
programs in applications of instructional technology. 

• Each program should encourage equal access for all pupils while allowing for uses 
targeted by grade, subject area, or special needs. 

Major Elements of AB 1470. AB 1470 continues portions of AB 803, the Educational Technology 
Local Assistance Program. It features, however, a significantly modified version of school-site grants, 
now known as School-Based Educational Technology Grants. AB 1470 added a new program of 
Research and Development Grants and continues funding for the six MTS Level I projects and six MTS 
Level II sites. The Act includes another new program category, the Regional Assistance Program, that 
re-authorizes the California Technology Project (CTP) and the regional ITV Agencies. The legislation 
reestablishes the Educational Technology Committee with increased representation from K-12 teachers, 
higher education faculty members, and business leaders. It also established the California Planning 
Commission for Educational Technology, responsible for developing a long-range educational 
technology master plan for the state. The master plan will include input from the Model Technology 
School (MTS) Projects, the Educational Technology Committee, the CDE, the California Post- 
Secondary Education Commission, and leaders in business and industry. 

AB 1470 also defines major areas of emphasis for programs prescribed in the bill including school-based 
technology use planning, coordination with state curriculum and instruction initiatives, effective staff 
development, and evaluation of the impact of the programs on teaching and learning. The evaluation 
component of AB 1470 was given greater emphasis with the recent signing of SB 1201 which amended 
AB 1470 by adding language. Senate Bill 1201 requires evaluation of the impact of the ITV agencies on 
local program implementation and directs that the results of the evaluation of AB 1470 programs be 
submitted to the California Planning Commission on Educational Technology. Figure 1, below, 
illustrates the major components of AB 1470. 

Priorities of AB 1470. In developing guidelines for implementation of AB 1470, the CDE developed 
priorities for the program based on a combination of legislative intentions and instructional initiatives 
established by the CDE. These state priorities, included in the guidelines for developing School-Based 
Educational Technology Grants (California Department of Education, 1990), are as follows: 

• Increase the use of coordinated school-based planning for the utilization of technology. 

• Ensure that school-level technology use is articulated with district programs and planning. 

• Facilitate the alignment and integration of technology into the curriculum. 

• Increase the access to and use of technology through effective management of learning resources. 

• Develop collaborative partnerships with business and industry and higher education. 

• Improve the quality and effectiveness of instruction and learning through the use of technology. 

• Increase levels of use of technology as a tool to improve instructional programs and practices. 

• Evaluate the impact of technology on teaching, learning, planning, and resource utilization. 
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One of the conditions for funding both the re-authorization of the existing educational technology 
programs and the establishment of new programs under AB 1470 was the "independent evaluation of all 
program components. The evaluation, funded with program funds, shall include procedures ... for the 
effective utilization of evaluation information to develop policy for use of educational technology" (Sec. 
51876.5 [e] EC Chapter 1334). The RFP which prompted this proposal was designed to fulfill this 
requirement and to provide additional evaluation information deemed necessary by the CDE. 

Addressing Legislative and CDE Priorities. In planning and conducting this study FWL and AIR 
focused upon the expressed concerns of the Legislative Analyst and the requirements for an independent 
state-wide evaluation of educational technology specified m AB 1470. The evaluation design 
emphasizes both formative assessment of program implementation and summative assessment of 
educational and program outcomes. Refinements of the evaluation plan were made in consultation with 
the Educational Technology Committee, the California Department of Education, and many others. 
Every effort was made to conduct the evaluation with consistency across projects and programs. The 
evaluation reports can be expected to serve as primary sources of information for the California Planning 
Commission for Educational Technology and the California Department of Education in developing the 
state master plan for educational technology. In general, the evaluation study should play a key role in 
shaping future policy, planning, implementation, and evaluation of educational technology in California. 



in. The Evaluation Plan 

Major Emphasis. The study of state-supported educational technology programs was designed to 
assess the impact of the programs in terms of implementation of program priorities established by the 
CDE and AB 1470. In the evaluation plan, these priorities have been categorized into a set of six areas 
of emphasis that were used in developing the general evaluation questions for both Phase I and Phase II. 

A. Six Priorities for Educational Technology Programs: 

L Site-Based Technology Use Planning: The degree to which educational technology is 
incorporated as an instructional strategy into school-level planning. 

2. Level of Implementation: The level of implementation of all six of the programs including 
supporting and impeding factors. 

3. Curriculum Support and Alignment: The degree to which the educational technology 
programs and applications are integrated with the State Curriculum Frameworks. 

4. Staff Development: The impact and availability of staff development made possible by the 
Educational Technology Local and Regional Assistance Programs. 

5. Learning Resources Management: The degree to which the programs facilitated the effective 
management of learning resources. 

6. Evaluation and Accountability: The degree to which formative and summative evaluation 
efforts were made by the six programs. 

Figure 2 illustrates the basic evaluation design. The Phase I and Phase II evaluated educational 
technology programs are listed on the vertical axis; the six state program priorities are listed on 
the horizontal axis. In addition to 19 AB 803 programs in Phase I there are six AB 1470 
programs to be evaluated in Phase II. The FWL-AIR evaluation provides information on 
outcomes both “across” the rows of the diagram for the effects of the various programs on each 
priority and “down” the columns of the diagram for the ways in which each of the state 
priorities was accomplished in each of the programs. For example, the School-Based 
Educational Technology Grant program was examined in terms of impact upon each of the state 
priorities, such as site-based planning, and was examined in terms of overall impact across all of 




3 



the priorities. In the same way, the priority on site-based planning was examined in terms of 
how it was accomplished in each of the six programs. The analysis of outcomes provides 
information of interest to stakeholder groups and policy makers including schools and school 
districts, regional agencies and consortia, the CDE, and the Legislature. 

Figure 2: Evaluation Design 



CDE and Legislative Program Priorities 
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Phase II Evaluation Plan: The six educational technology programs have undergone intensive study 
in Phase n. The evaluation methodologies were designed to address questions that are unique to the 
objectives of each program and questions related to CDE and legislative priorities. Phase II provides 
intensive assessment, including information and data on the needs, objectives, and major state-level 
activities addressed by each program as well as responses to specific evaluation questions. There are 
three categories of questions to be addressed in Phase II: general questions, project-specific questions, 
and policy questions. The general evaluation questions addressed in Phase II include the 14 stated in the 
RFP plus additional questions identified by the evaluation staff. A set of evaluation questions was 
devised and categorized according to the initiatives and priorities for assessment. The instrumentation 
used to gather the information needed to answer each of the questions is shown in the project-specific 
instrumentation matrices in each of the six sections of this report. A copy of each assessment instrument 
is included at the end of each section of the report. 
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Assessment Procedures: Assessment instrumentation was designed to answer each of the evaluation 
questions for each of the six programs studied. Most of the instruments are being adapted for additional 
future use and will be included as the major component of the Phase IV section of this project. All 
instruments were designed with input from staff of the projects being evaluated, the CDE Office of 
Educational Technology, and the CETAP Project Advisory Committee. Instrumentation was field 
tested, reviewed, edited, and administered to the selected target populations. 

Details regarding each assessment instrument and its specific questions are incorporated into the report 
for each of the six programs. Procedures and schedules of the evaluation administration are also 
discussed in each section. Copies of each of the CETAP Assessment Instruments are included as the 
appendix for each of the six project components. The following pages include selected references used 
as supplementary information sources for this report. All assessments were administered during the 
spring of 1991. Table one lists the instrumentation and assessment procedures with the number of 
planned recipients and the number who actually completed the assessment for each of the six programs 
studied. 



Table 1. Assessment Procedures and Number of Assessments Planned and 
Completed for the CETAP 



Phase n Assessment Procedures 


Number 

Planned 


Number 

Completed 


1.0 School-Based Educational Technology Projects 






Self-Assessment Inventory 


367 


285 


Telephone Interview Questions 


30 


30 


Teacher Survey of Technology Use 


500 


319 


Student Survey of Technology Use 


700 


523 


Site Interviews and Observations 


18 


18 


2.0 Level I Model Technology Schools (AIR Subcontract) 






Self-Assessment Inventory 


6 


6 


Teacher Survey 


1091 


568 


Elementary Student Questionnaire 


- 


1060 


Secondary Student Questionnaire 


- 


1269 


Site Interviews and Observations 


6 


6 


Visitor Survey 


• 


357 


3.0 Academic Technology Model Programs (Level II) 






Self-Assessment Inventory 


6 


6 


Site Interviews and Observations 


6 


6 


Teacher Survey of Technology Use 


65 


53 


Student Survey of Technology Use 


1,000 


800 


4.0 Instructional Television 






Self-Assessment Inventory* 


7/7 


7/6 


Assessment of ITV and Media Services 


70 


39 


CPB Survey: California Addendum-District Office 


200 


138 


CPB Survey: California Addendum-Principals 


522 


419 


CPB Survey: California Addendum-Teachers 


1,035 


632 


CPB Survey: All items for District, Principals, Teachers 


1,757 


1,757 


5.0 Software Development Projects 






Self-Assessment Inventory 


6 


5 


Software User Survey 


100 + 


33 


6.0 California Technology Project 






Self-Assessment Inventory* 


14/14 


13/13 


Assessment of Services 


900 


311 


Telephone Survey 


14 


14 


Other Data Collection 






Follow-up telephone survey of summer institute participants 


73 


73 



* Inventories collected for 1989-90 and 1990-91 
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